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ABSTRACT : 

Two  formulae  are  derived  for  ratios  of  limiting  probabilities  for 
a  class  of  finite  homogeneous  Markov  chains.   The  class  consists  of  chains 
obtained  by  generalization  of  Bernoulli  random  walk  with  reflecting  or 
absorbing  barriers.   These  chains  are  closely  related  to  problems  of 
testing  hypotheses  with  finite  memory.   The  formulae  are  recursive  in 
nature  and  hence  much  easier  to  use  than  classical  methods. 


Prepared  by: 


1.   Introduction. 

In  this  report  we  propose  a  method  for  recursive  evaluation  of 
certain  probabilities  associated  with  two  classes  of  finite  homogeneous 
Markov  chains.   These  chains  are  next  of  kin  to  Bernoulli  random  walks 
with  reflecting  and  absorbing  barriers.   They  may  be  roughly  character- 
ized by  the  following  four  properties: 

(1)  Except  for  the  barrier  states,  (one-step)  transitions  from  each 
state  can  be  made  to  exactly  two  other  states. 

(2)  The  states  are  divided  into  two  subsets,   S   and  S  ,   plus  an 
initial  state  in  the  absorbing  case.   States  in  S   communicate 
with  states  in  S   only  via  a  pair  of  states,  one  in  each  subset. 

(3)  Except  for  the  absorbing  states,  the  probability  of  transition 
among  states  in  S   and  among  states  in  S   has  only  two  values, 
p  ,   1  -  p   and  p  ,   1  -  p   respectively. 

(4)  If  states  in  each  of  the  two  subsets  are  ordered  such  that  the 
transitions  with  probability  p  are  all  between  adjacent  states  in 
one  direction  then  the  transitions  with  probability   1  -  p  are  all 
in  the  opposite  direction,  but  not  necessarily  to  the  adjacent  state. 

A  glance  at  Figures  1  and  3  may  help  to  reveal  the  structure  of  a  typical 

member  of  those  two  classes. 

We  are  interested  in  the  ratio  of  limiting  probabilities  of  a 

chain  being  in  the  subset  S   and  the  subset  S,  .   In  sections  2  and  3 

a  b 

we  present  a  recursive  formulae  for  evaluating  these  probabilities.   Subse- 
quent examples  show  that  the  computation  is  considerably  simpler  than  the 
classical  method  of  solving  systems  of  linear  equations.   Our  method  involves 
nothing  but  repeated  substitution  and  Is  easy  to  perform  and  program  even 
for  a  large  number  of  states. 

The  need  for  studying  these  ratios  arises  in  problems  connected 
with  finite  automata  with  binary  inputs  and  outputs  driven  by  a  Bernoulli 
sequence.   These,  in  turn,  appear  in  the  so-called  finite  memory  problems 


(References  [1]  through  [4]),  which  are  currently  receiving  considerable 
attention  in  literature. 

The  reason  for  writing  this  report  is  twofold.   First  since  the 
proofs  of  our  formulae  (Sections  4  and  5)  are  basically  algebraic  and  thus 
rather  long  it  is  usually  necessary  to  condense  the  proof  when  the  formula 
is  used  as  a  lemma.   Hence  we  wanted  to  have  the  proof  documented  in  full 
detail  for  reference.   Next,  it  is  conceivable  that  Markov  chains  of  the 
type  studied  here  may  be  encountered  in  various  stochastic  models.   Hence, 
the  second  purpose  of  this  report  is  to  provide  an  access  to  our  results 
to  other  workers  in  the  general  area  of  stochastic  modelling. 

To  this  we  would  like  to  add  that  the  two  formulae  can  probably  be 
generalized  in  several  directions.   For  instance,  inspection  of  the  proofs 
indicate  that  the  same  method  could  still  be  used  to  establish  similar 
formulae  for  a  larger  class  of  chains,  namely  without  the  property  (3) 
above. 

The  part  on  ergodic  chains  (Sections  2  and  4)  and  the  part  on 
absorbing  chains  (Sections  3  and  5)  can  be  read  independently. 


2.   Ergodic  Chains. 

Let   r  =  {r  (2)  ,r  (3),...}   and  r,  =  {r,  (2)  ,r,  (3)  , . .  .  }   be  two 
— a     a     a  — b     b     b 

sequences  of  positive  integers  such  that  1  ^  r  (i)  <  i,   1  £  r,  (i)  <  i, 

a  b 

i  =  2,3,...   With  each  such  pair   (r  ,r,  )   we  associate  a  class 

—a  — b 

E(r  'ij  =  M  :  n=l,2,..,  ;  m=l,2,...} 

— a  — b      n,m 

of  finite  ergodic  Markov  chains.   The  chain  M     has  n  +  m  states 

n,m 

which  are  divided  into  two  subsets   S   and   S.   with  n  and  m  states 

a        b 

respectively. 

We  label  the  states  in  S   by   (i,a) ,   i  =  l,...,n,   and  the 

a 

states  in  S,   by   (i,b) ,   i  =  l,...,m.   The  transition  probabilities  are 
b 

as  follows: 

P((i,a)  ->  (i+l,a))  =  p  ,    i  =  l,...,n-l, 

P((n,a)  ■+  (m,b))    =  p 

P((i,b)  -y   (i+l,b))  =  pb,    i  =  l,...,m-l, 

P((m,b)  ^  (n,a))    =  pb, 

P((i,a)  ^  (r  (i),a))  -  q  ,    i  =  2,...,n, 

P((l,a)  +   (l,a))    =  q  , 

a 

P((i,b)  ->  (rb(i),b))  =  qb,    i  =  2,...,m, 

P((l,b)  ->  (l,b))    =  qb. 

Here  0<p  <1,   0  <  p,  <  1,   q  =l-p,   q.  ■  1  -  p.  .   All  other 
Ka  b        a        a    b        b 

transition  probabilities  are  zero.   The  transition  diagram  is  depicted  in 
Figure  1. 


<S5E* 


Figure    I 


Proposition  1:   Let   M    6  E(r  ,r  ) ,   let   m(s),  s  £  S   US,   be  its 
c n,m     — a  — b  a    b 

stationary  distribution,  let 


u(S  )  =    J  u(s)    and    u(S,  )  =   £  u(s) 

a    s€S  b    s6Su 

a  b 

be  the  stationary  probabilities  of  the  chain  being  in  S    and   S 

a        b 


respectively.   Then 


li(S  )    p™  A 
a     b    n 

y(S,  )     n  B   ' 
b     p     m 
a 


(2.1) 


where  A   and   B    are  polynomials  in   p   and   p,   respectively  satisfying 
n        m  ra        b  ' 

the  recurrence  relations 


Vl'»I+V         LiAPr">      Al  =  1'  n-1,2,....  (2.2) 

l-r    (n+1) 
a 


m 

vi  =  pr+% ,  i  *& .."i-1-    m  =  i>2--- •    <2-3) 

£=r    (m+1) 
b 

Hence,  both  A   and   B    have  integral  coefficients  and  are  of  degree 
n        m 

less  than  n   and  m,   respectively. 
(For  the  proof  see  Section  4.) 


Example  1:   Let   (r  , r,  )   be  given  by  the  following  table 
—a  — b 

i r  O) r,(i) 

a b 


2  1  1 

3  1  1 

4  2  3 

5  •  3 


Figure    2 


let  n  =  A,   m  =  5.   The  transition  diagram  of  this  9-state  chain  is 

in  Figure  3. 

First  evaluate  A,   and  Br.   From  (2.2)  we  have 
4        5 

A3  =  Pa  +  W."'  +  "aVa^  ■ 


A   =  p  +  q  A  p  1_1 
2    a    a  la 


A1  =  1  , 


and  substituting  from  the  bottom  to  the  top  gives 


A-l  =  1  , 


A2  =  Pa  +  qa  =  X  ' 


A  =  p2  +  q  p  +  q   , 
3    a    a  a    a 


A.  =  p3  +  q  p  +  q  p2  +  q2p  +  q2  , 
4    a    a  a    a  a    a  a    a 


or  by  substituting   q  =  1  -  p 

3.  a. 


A  =  p3  -  p2  +  1 
4    a    a 


Similarly  from  (2.3) 


Bs  ■  K  +  Wr^O  • 


B4  "  pb  +  qbB3pb"3  ' 


b3  ■  pb  +  %(Bipri+B2pr2)  • 


B2  -  pb  +  \*A~l 


■l-1' 


and  again  substituting 


Bx-1    . 


B2    -    P,   +    qb    -    1    , 


B3  =  Pb  +  qbpb  +  qb    ' 


B4    =    Pb   +    Vb   +   qbPb   +    qb    ' 


B5  =  Pb  +  Vb  +  qbPb  +  qbPb 


+  ^uPu  +  q^p?  +  q^Pu  +  q? 


Vb  T  Vb  T  Vb  T  4b  ' 


or  substituting  for   q  =  1  -  p 

b        b 


Hence,  from  (2.1) 


B5  -  p3  -  pb  +  1. 


"<s«>   Pb  "«  -  Pa  +  X 
"<sb>  "pI  Pb  -  Pb  +  1 


3.   Absorbing  Chains. 

Let  r  =  {r  (l),r  (2),...}   and  r.  =  {r,  (1)  ,r,  (2)  , . . .}   be  two 

cl        a  3.  ~ ~U         D         D 

sequences  of  nonnegative  integers  such  that 


0  <:  r  (i)  <  i,   0  £  r,  (i)  <  i,     i  =  1,2,... 
a  d 

With  each  such  a  pair   (r  ,r,  )   we  associate  a  class 

-~ a  — b 

A(r  ,r_)  =  {M    :  n=l,2,...  ;  m=l,2,...} 
— a  — b      n,m 

of  finite  absorbing  Markov  chains.   The  chain  M     has  n  +  m  +  1   states, 

n,m 

two  of  them  absorbing  and  the  rest  transient.   One  state  is  always 

designated  as  an  initial  state  while  the  remaining  n  +  m  states  are 

divided  into  two  subsets   S   and  S,   with  n  and  m  states  respectively, 

a        b 

each  containing  one  of  the  two  absorbing  states. 

We  label  the  states  in  S   by   (i,a)   i  =  l,...,n  with   (n,a) 

absorbing,  and  the  states  in  S,   by   (i,b)  ,   i  =  1 , . . . ,m  with   (m,b) 

b 

absorbing.   The  initial  state  is  labeled   (0,a)   or   (0,b)   or  just  0 
as  needed. 

The  transition  probabilities  are  as  follows: 

P((i,a)  +    (i+l,a))  =  p  ,       i  =  l,...,n-l, 

a, 

P((n,a)  +  (n,a))  =  1, 

P((i,b)  ■>  (i+l,b))  =  pb,       i  =  l,...,m-l, 

P((m,b)  ->  (m,b))  =  1, 

P((i,a)  ->  (r  (i),a))  =  q  ,      i  -  l,...,n-l, 

3.  3l 

P((i,b)  ■*   (rb(i),b))  =  qb,     i  =  l,...,m-l, 


10 


Pa 

P(0  ■+  (l,a))  = 


Pa  +  Pb  ' 


Pb 


P(0  ->  (l,b))  =  —J 


Pa  +  Pb 

Here  0<p  <  1,   0  <  p,<  1.   q  =l-p,   q.  =  1  -  p..   All  other 
a  b        a        a    b        b 

transition  probabilities  are  zero.   The  transition  diagram  is  depicted 
in  Figure  2. 

Proposition:   Let  M    6  A(r  ,r.  ),   let  Tr(a)   and  Tr(b)   be  the  probabilities 
* n,m     —a  — b 

of  absorption  in  the  state   (n,a)   and   (m,b)   respectively,  if  the  initial 
state  is  the  state  0.   Then 

7T(b)     m  A   '  U>i) 

pb   n 

where  A   and  B   are  polynomials  in  p   and  p   respectively  satisfying 

the  recurrence  relations 

n         k-r  (k) 
An+1=l-q   I     Ar    pa  a    .  AQ=0,     n-1.2.....     (3.2) 
k=2   a 


m         k-r  (k) 

B«u  =  1  -  %  X  \ck)pb     •  Bo  =  °-    m  =  1'2 (3'3) 

k=2   b 


Hence  both  A   and  B   have  integral  coefficients  and  are  of  degree 
n        m 

less  than  n  and  m   respectively. 
(For  the  proof  see  Section  5.) 


11 


INITIAL 


12 


Example  2 :   Let   (r  ,r_  )   be  given  by  the  following  table 


i 

r    (i) 

r.  (i) 

1 

a 

0 

u 
0 

2 

1 

1 

3 

1 

1 

4 

2 

3 

5 

• 

3 

let  n  =  4,   m  =  5 .   The  transition  diagram  of  this   10-state  chain  is 
in  Figure  4. 

First  evaluate  A.   and  Br.   From  (3.2)  we  have 

4        j 

\  "  1   -   Wa^Va"^  ' 

ai  - r  -  %vr°  • 


AQ  =  0  , 


and  substituting  from  the  bottom  to  the  top 


AQ  =  0  , 


A1  =  1  , 


A   =  1  -  q  p   MP   > 
4        a  a    a  a 


or  substituting  for   q  =  1  -  p 

a        a 


A,  =  P3  -  P  +  1 
4    a    a 


Similarly  from  (3.3) 


S'1-  ib<Vb",+BiI'b"1+B3I,b"l) 

Bl  =  1' 


13 


Figure    4 


and  again  substituting 


or 


Hence,  from  (3.1) 


Bl  =  1  ' 


B3  =  X  "  Vb  ' 


b5  -  i  -  pbqb  -  ?l%  -  pbqb  +  p^l  . 


Bc  =  2pJ  -  pj  -  pu  +  1 


it  (a)    Pa  2pb  "  Pb  "  Pb  +  X 

.(b)    -p|       pj-.p.+  l 


14 


15 


4 .   Proof  of  Proposition  1. 

Let  P  be  the  transition  probability  matrix  for  the  chain  M   , 

n  ,m 

where  the  first  n  rows  and  columns  correspond  to  states   (1 ,a) , . .  .  ,  (n,a) 
and  the  following  m   rows  and  columns  to  states   (m,b) , . . . , (1 ,b) . 

Let   y_  =  (y(l,a)  , .  .  .  ,u(n,a)  ,u(m,b)  , .  .  .  ,u(l,b))   be  the  stationary 
distribution,  so  that 


M(I-P)  =  0  , 


(4.1) 


where  I   is  the  identity  matrix.   Now  partition  the  matrix  P   into  four 
submatrices 


P  = 


P     V 
a     a 


V     P 
b     b 


where  P   is  an  n  x  n  matrix 
a 


Row: 


P   = 
a 


r  (i) 
a 


Column: 


ra(i) 


a 


and     P,       is   an     m  x  m     matrix 
b 


Row: 


16 


pb  = 


ra(1) 


Column: 


n 


rb(i) 


Notice  that  each  row  of  these  matrixes  contains  exactly  one  entry  q, 

namely  the   (i,r(i)) —  one,  and  that  the  labelling  of  rows  and  columns 

of  P,   begins  at  the  lower  right  corner  while  the  labelling  of  P   is 
b  a 

the  usual  one  (beginning  at  the  upper  left  corner) . 

The  off-diagonal  matrices  V   and  V,   consist  of  all  zeros  except 

a        b 

for  the  lower-left  corner  entry  of  V  ,  which  is   p  ,   and  the  upper- 

cl  3. 

right  corner  entry  of  V,  ,   which  is   p,  . 

b  b 

With  this  partitioning  the  equation  (4.1)  decomposes  into  two 
equations 

u^(I-Pa)  =  (0,...,0,y(m,b)pb),  (4.2) 


u^(I-Pb)  =  (y(n,a)pa,0,...,0), 


(4.3) 


where 


17 


u  =  (y(l,a) ,...,u(n,a))  , 


and 


Ji,  =  (u(m,b)  ,.  ..  ,y(l,b))  . 


■R) 


Consider  the  matrix  equation  (4.2)  first.   Solving  for  \i        gives 


—a 


y^  =  (0,...,0,u(m,b)pb)(I-Pa)  X  , 

or  denoting  a..   the   (i,j) —  entry  of  the  inverse   (I-P  )~ 
ij  a 


u(i,a)  =  p(m,b)pba    ,     j  =  l,...,n. 


(4. A) 


By  the  well-known  formula  for  matrix  inversion 


I-P 


a .  . 

id 


(j,i) 


I-P 


where    I-P     is  the  determinant  of   I-P   and    I  -  P   ,,  „s 
1      a  '  a       '      a1  (k,£) 

t*  Vi 

is  the   (k,&) —  cofactor  of    I-P   I  .   Hence 

1      a  ' 

n  y(m,b)p     n 

U(8a)  =  h(i,a)  "  1  I  -  Pa  1  J,  I  1-\    I  (La)" 


(4.5) 


Next  let  A   be  the  determinant  of  the  n  x  n  matrix  obtained  from 
n 


I-P   by  replacing  the 

3, 


th 


column  by  a  column  of   l's, 


13 


A  = 
n 


"<L 


(4.6) 


Expanding  A   along  this  last  column  we  obtain 
n 


LI 

^     =  l        l    ~   P  \n 
n   ,*•_  '      a  '  (i.n 

i=l 


)' 


th 


n 


since  the   (i,n) —  cof actors  of  A   and    I  -  P     are  identical. 
Thus,  (4.5)  can  be  written  as 


u(m,b)p 

"oy =  i  i-p  i  \  • 


Next  since  the  only  flow  of  probability  between  sets   S   and  S   is 
through  states   (n,a)   and   (m,b)   we  must  have 


y(m,b)p  =  y(n,a)p 
b  a 


(4.7) 


in  the  stationary  regime.   Using  (4.4)  we  obtain 

I  I  -  P 


y(n,a)  =  y(m,b)p 


a  ' (n,n) 


I-P 


and  substituting  from  (4.7) 


But 


I-P    =  p   I  -  P    .   . 
a  '    ra'      a  ' (n,n) 


19 


I-P 


a  '(n,n) 


which  is  same  as  the  determinant  of  the   (n-1)  *  (n-1)   matrix  I-P 

a 

obtained  for  the  chain  M      £  E(r  ,r_ ) .   Employing  temporarily  the  super- 
script  (n)   for  the  number  of  states  in  S   we  have  a  recurrence  relation 

a 


z(n)  _  p(n) 

a 


-  P  I  i'"-1'  -  p<°-1> 

a '  a 


and  since    I-P      =  p   we  obtain 
1        a   '    a 


I<n>  _  P(n>  I  =  pn, 
a    '     a 


Thus 


U(S  )  =  -~  P(m,b)p  . 
an       d 
P„ 


Now  going  back  to  (4.3)  and  repeating  all  the  steps  above  we  obtain  a 
similar  expression 


20 


B 


m 


y(S,  )  =  — -  y(n,a)p  , 
Dm        a 

Pb 


where  B   is  the  determinant  of  order  m 
m 


m 


Hence,  using  again  (4.7)  we  have 

U(S  )  p"  A 

a  b   n 

y(S  )  n  B   ' 

b  p    m 
a 

and  it  remains  to  prove  that  A  and  B   satisfy  the  recurrence  relations 

n  m 


(2.2),  (2.3). 


let 


We  begin  with  the  determinant  A  .   To  evaluate  this  determinant 

n 


I1  =  {i  =  2,...,n  :  ra(i)  =  1} 


Notice  that   I..   is  the  set  of  exactly  those  row  indices   i   for  which  the 
(i,l) —  entries  in  (4.6)  are  -q  .   Now  multiply  the  first  row  in  (4.6) 
by  q  /p   and  add  it  to  all  rows  such  that   i  €  I, •   The  determinant 

3.    3.  -L 


becomes 


21 


P  "P 
0  p  -p 

-q  i  -p 


n 


1  -p 


-q 


.(n) 
12 


1  -p 


ui 


'12 


where  we  temporarily  dropped  the  subscript   a   to  ease  the  notation, 
entries  in  the  last  column  are  given  by 


The 


(n)  _ 
'2i 


1  +  q/p   if   i  €  I1, 


if  i  £   I1- 


Expanding  this  determinant  along  the  first  column  we  have 


where  D 


(n) 


4a)  -  PD{n) 


A   is  the  original  determinant  (4.6)  and 
n 

t(n) 

22 


,(n) 


(n) 

2n 


22 


is  of  order  n  -  1.   Notice  that  the  entries  in  the  first  column  in  D 


(n) 


are   -q  only  for  row  indices   i  =  3,...,n  such  that  either   r(i)  =  1 
or  r(i)  =  2.   Hence,  calling  I   =  {i  -  3, . . . ,n  :  r(i)  <  3},   multiplying 
the  first  row  in  D^    by  p/q  and  adding  to  rows  with   i  (j  I   this 
determinant  becomes 


P  "P 
0  p  - 


1  -p 


.00 

'32 


"2n 


The  entries  in  the  last  column  are 


#- 


t(n)  +  P_  t(n)   if  ± 

Z2±         q  22   "   1  fc  V 


t(n) 
2i 


if  i  6   I2- 


Expanding  again  along  the  first  column  we  have 


,(n) 


=  PD 


(n) 


where 


,(n) 


P  "P 


1  -p 


-q 


l  -P 


(n) 
'33 


23 


Proceeding  in  this  fashion  we  obtain  a  sequence  of  determinants 


D(n)   (n)      (n) 
1  ,U2      un  * 


(A. 8) 


where  D  n   is  of  order  n  -  k  +  1.   The  entries   t,  .    in  the  last 
K  ki 

column  of  D     satisfy  the  recurrence  relation 


where 


.00 

'k+1 ,  i 


tw)+?tv?      if     i€V 
ki  q     kk  k 


'ki 


if   i^Ik, 


(n) 


k  =  1, . . . ,n;   i  =  k,. . . ,n,   ^ 


=  1, 


I,  =  {i  =  k+l,...,n  :  r(i)  £.   k} 


(4.9) 


Further 


so  that 


Dk+1  "  pDk   ' 


n(n)  n-1  (n)  _  n-1  (n) 

D,    =  p    D  =  p    t 
1          n  nn 

Consider  now  the  determinant  A       of  order  n  +  1  obtained 


(4.10) 


by  using  the  same  sequence  _r, 


(n+1) 
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Applying  the  above  procedure  to  D      =  A       we  obtain  a  sequence 


,(n+l) 
1 


d;~  '.,..., D 


(ttfl) 

n+1   » 


(A. 11) 


where  the  determinants  D,       again  satisfy  (4.9)  with  n  replaced  by 
n+1.   Arrange  now  the  last  columns  of  the  sequences  (4.8)  and  (4.11) 
into  triangular  arrays  as  follows : 


(n) 
■11 
.(n) 
'12 


22 


,(n) 


(n)    (n) 
In  '  C2n 


'nn 


(n+1) 


.(n+1) 
11 

(n+1)  (n+1) 
12   '22 


(n+1)  (n+1) 
In   '  2n 

(n+1)  (n+1) 
l,n+r  2, n+1 


(n+1) 


nn 


(n+1)  (n+1) 
n,n+l'  n+1, n+1 
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Since  for   i  s£  n  by  the  definition  of  sets   I, 

k 

i  €  l£n)   if  and  only  if   i  €  I^*"0 

the  first  n  rows  of  T     and  T       are  identical,  i.e. 

(n)     (n+1)       .  .  //  -,^\ 

Tel    Tel    '     i=k,...,n;   k=l,...,n.  (4.12) 

Next  by  (4.9) 

1  if   k  =  l,...,r(n+l) , 


t(n+l)  =  \ 


-k,n+l    j 


1  +  £    I  t.(r^1)   if  k  =  r(n+l)+l,...,n+l. 


P  £=r(n+l)  l'1 


In  particular  for  k  =  n  +  1   since  r(n+l)  <  n  +  1 


n 


tSFL -i  +  *      I    ^+1)-  (*•«) 


"n+l.n+l       p  t=r(J;+1)  M 


But  by  (4.12)  for   £  <  n  +  1 


(n+1)     (n)       m      (I) 


so   that    (4.13)    becomes 


(nfl)        .4  8         U) 

Vl.n+1  p  ^j   U 


Hence,   by    (4.10) 


p  v   £=p(n+l)    p 
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or  calling  again  D    =  A   we  have 


n 
n  ,        r      n-Jl 


An+1  =  Pa  +  qa      I        V  '   '      n  =  1»2»'"> 
n+i    a    a  A=r(n+1)  * 


where  clearly  A  =  1. 

The  recurrence  relation  for  B   is  established  in  exactly  the 

m 

same  fashion. 

Noticing  the  obvious  fact  that  the  polynomials  A   and  B   must 
have  integral  coefficients  completes  the  proof  of  Proposition  1. 


27 


5.   Proof  of  Proposition  2. 

Notice  first  that  with  0  being  the  initial  state  any  subsequent 

visit  to  this  state  is  a  recurrent  event.   Call  this  event  E  .   Next  call 

E    the  event  which  occurs  if  the  chain  after  leaving  the  state  0   reaches 
a 

the  absorbing  state   (n,a)   without  any  further  visit  to  state  0.   Similarly, 

define  E,   for  the  absorbing  state   (m,b) .   Now  clearly 

b 

pcy  >  o, 

and  since  the  absorbtion  in   (n,a)   occurs  if  and  only  if  we  have  either 


E   or  R^E   or  R^R~E    etc. 
a       0  a       0  0a 


P(Ea) 
tv  (a)  = 


1  -  P(EQ)  ' 


and  similarly 


so  that, 


P(Eb) 
*(b)  =  1  -  P(EQ)   » 


(    \         P(E  > 
Tr(a)      a 

iT(b)  "  P(Eb)   * 


(5.1) 


Next 


P(E  )  =  1 PCE1),  (5.2) 

a    d  +  P,     a 


a    Pa  +  Pb 


where  E1   is  the  event  which  occurs  if  and  only  if  the  chain  after  leaving 
a 

the  state   (l,a)   reaches  the  absorbing  state   (n,a)   without  ever  visiting 

the  state  0. 

Consider  now  a  subchain  Ma  obtained  from  the  chain  M     by 
making  the  state  0  an  absorbing  state  and  deleting  states   (l,b)   though 


28 


(in,b)  .   The  transition  probability  matrix  for  this  subchain  is  the 
(n+1)  x  (n+1)   matrix 


All  O's 


If  this  subchain  is  started  at  the  state   (l.a)   then  PCE1)   is 

a 

equal  to  the  probability  of  absorption  in   (n,a)   for  this  subchain.   Using 
the  well-known  result  from  the  algebraic  theory  of  Markov  chain  (cf.  [5], 
Theorem  3.3.7)  we  have 


PCE1)  =  p  ttl 

a    ra  l,n-l 


(5.3) 


-1 


where   [a..]  =  (I-Q  )    and  Q   is  the   (n-1)  x  (n-1)   matrix  of  transi- 
ij       a  a 

tion  probabilities  between  transient  states  of  M  ,   i.e. 
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All  O's 


By  the  formula  for  matrix  inversion 


a1 (n-1,1) 


l,n-l     I  -  Q 


(5.4) 


where    I  -  Q    is  the  determinant  of   I  -  Q   and 
■a1  a 

st 
is  its   (n-1,1) —  cofactor.   Now 


I  -  Q 


a1 (n-1,1) 


|l-Q  L   .  n  =  (-D 
1   *a ' (n-1,1) 


"P. 


-q     1  ~P 
^a      v  a 


All  O's 


-<1. 


1  -F 


■   P. 


n-2 


-q. 


X-Pa 


and  calling  A 


I  -  Q     we  obtain  from  (5.2),  (5.3)   and  (5.4) 


n 
ra    b  n 


(5.5) 
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Following  the  same  procedure  for  the  event  E,   we  obtain 


m 


P(EL)  =  ^ L 

v  t>    p  +  pv  B   » 
a    rb  m 


(5.6) 


where  Bm  =  I  !  ~  Qb  I   and  Qb   is  the   (m-1)  x    (m-1)   matrix  of  transi- 
tion probabilities  between  transient  states   (l,b) , . . . , (m-l,b) .   Hence, 
(5.1),  (5.5)  and  (5.6)  yields  (3.1)  and  it  remains  to  establish  the  recur- 
rence relations  for  A   and  B  . 

n        m 

To  do  this  we  evaluate  the  determinant 


1  -p 


All  0's 


1  -p 


-q 


l  -P 


-q 


where  we  dropped  the  subscript  a   to  ease  the  notation.   Notice  that  A 

is  of  order  n  -  1  and  that  each  row  but  the  first  has  either  exactly  one 

subdiagonal  entry  equal  to   -q,   namely  the   (i,r(i)) — ,   or  all  subdia- 

gonal  entries  are  zero.   The  former  case  occurs  if   r(i)  >  0  while  the 

latter  if  r(i)  =  0. 

Next  consider  the  determinant  A  ,  ..   of  order  n  obtain  for  the 

n+1 

chain  M  ,  .,    with  the  same   (r  ,  r,  )  .   Then 
n+l,m  -a  -=13 


n+1 


n 


-q 
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Now  if   r  (n)  =  0   then  there  i 
a 


s  no   -q  in  the  last  row  and  hence 


A    -  A  . 
n+1    n 


(5.7) 


If   *  (n)  >  0   then  expanding  A     along  the  last  col 


n+1 


.umn  gives 


A    =  A  +  pD  , 
n+1    n   K  n 


(5.8) 


where 


n 


0 

# 

A   1 

n-1 

• 

0 

-p 

-q                1 

0 

is  of  order  n-1.   Notice  that  the  entry  -q  in  the  last  row  moved  one 

step  to  the  right.   Expanding  D   again  along  the  last  column  gives 

n 


n     n— 1 


where  D   ,   is  of  order  n  -  2 
n-1 


n-1 


n-2 


-q 
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Now  repeating  this  the  entry  -q  eventually  (after  n  -  r  (n)   steps) 
reaches  the  diagonal  and  we  have 


r  (n)+l 
a 


V(n) 

a 


O'-q 


and  expanding  this  determinant  along  the  last  row  yields 


D 


r  (n)+l 


=  -qA^ 


a  a 

Substituting  back  into  (5.8)  we  obtain 


(n)* 


.       .      n-r(n) 
A  ,  =  A  -  qp       A  ,    v 
n+1    n  r  (n)  , 


(5.9) 


which  holds  for  any  n  =  1,2,...   such  that 


r  (n)  >  0. 
a 
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To  include  the  case   r  (n)  =  0  define  An   =  0 .   Then  (5.9)  reduces  to 

a  U 

(5.7).   Finally,  repeatedly  substituting  for  A   in  (5.9)  and  using  the 

n 

obvious  fact  A  =  1  we  obtain  the  recurrence  relation  (3.2).   Notice 
that  (3.2)  holds  also  for  n  =  1   since  r  (1)  =  0  always. 

cL 

The  relation  (3.3)  for  B   is  established  in  exactly  the  same 

m 

fashion  from    I  -  Q,   . 

b  ' 

Noticing  the  obvious  fact  that  the  polynomials  A   and  B   must 
°  y  n        m 

have  integral  coefficients  completes  the  proof  of  Proposition  2. 
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